Skip to content

fix(release): keep the release a draft - the uploader was publishing it - #70

Merged
MotherSphere merged 1 commit into
mainfrom
fix/release-draft-hold
Aug 27, 2026
Merged

fix(release): keep the release a draft - the uploader was publishing it#70
MotherSphere merged 1 commit into
mainfrom
fix/release-draft-hold

Conversation

@MotherSphere

Copy link
Copy Markdown
Member

A pre-flight review of the release pipeline, run before merging the 0.10.0
release PR, found that the draft hold shipped in #68 does not work. This fixes
it before it ever runs.

The bug

softprops/action-gh-release calls finalizeRelease() after uploading, which
PATCHes draft: false. Its only guard is
input_draft === true || release.draft === false, and an omitted draft: input
parses to undefined, so neither disjunct fires. The action documents this in
its own action.yml:

When reusing an existing draft release, set this to true to keep it draft;
omit it to publish after upload.

What would have happened on merge of #69

The first build leg to finish uploading publishes v0.10.0 as /releases/latest
carrying one of four binaries and zero signatures. It stays public for the
remaining three builds and the entire sign job. The sign job's closing
gh release edit --draft=false succeeds as a no-op, so nothing reports a
problem and every check goes green.

Self-update is fail-closed on .sig, .meta and .meta.sig, so every running
Colony would have been offered an update that cannot be applied. If a build leg
or the sign job then failed, the release stays permanently public, partial and
unsigned — the v0.7.0 incident, reached automatically, while the workflow
comment assures the operator that state cannot exist.

The fix

draft: true on the upload step, plus a following step that asserts the hold
survived — so a future action bump cannot reintroduce this silently, and it
fails the build leg rather than the release.

Two more real issues found in the same review:

  • The signing key was removed with a trailing rm, which bash -e skips when
    sign-release.sh exits non-zero, leaving the PEM on the runner for the rest
    of the job. Now a trap ... EXIT.
  • The AUR host-key cross-check reported "does not match the pinned one" for any
    network failure: ssh-keyscan's empty output went straight into grep and
    its exit code was discarded. An alarming message for the wrong reason is worse
    than none. An unreachable host now warns; a genuinely different key still
    stops the release.

And a documentation correctness fix that matters more than it looks:
docs/release-signing.md told a maintainer to run sign-release.sh on local
files. Rust release builds are not bit-reproducible, and the script hashes
whatever it is handed into the .meta sidecar, which the client then enforces
against the bytes it downloaded. Signing a fresh cargo build --release would
produce a release that fails verification for everyone — worse than
unsigned, because it also fails closed. It now downloads the published assets
first, and carries a runbook for both failure states, including the trap that
"Re-run all jobs" reports green while skipping every downstream job.

The same draft: true fix has been applied to the canonical template in
Project-Colony-Resources, which had inherited the bug.

The draft hold added last commit does not work, and a pre-flight review of the
pipeline caught it before the first release ran.

`softprops/action-gh-release` calls `finalizeRelease()` after uploading, which
PATCHes `draft: false`. Its only guard is
`input_draft === true || release.draft === false`, and an omitted `draft:` input
parses to `undefined`, so neither disjunct fires. The action's own action.yml
states it plainly: "When reusing an existing draft release, set this to true to
keep it draft; omit it to publish after upload."

So the first matrix leg to finish uploading would have published v0.10.0 as
/releases/latest carrying one of four binaries and zero signatures, left it
public for the remaining builds and the whole sign job, and made the sign job's
closing `--draft=false` a silent no-op. Every check would have been green. If a
build leg or the sign job then failed, the release would have stayed permanently
public, partial and unsigned - the v0.7.0 incident, reached automatically, with
the workflow comment assuring the operator that state could not exist.

`draft: true` on the upload step fixes it. A following step asserts the hold
survived, so a future action bump cannot reintroduce this silently, and it fails
the build leg rather than the release.

Also in this commit:

- The signing key is removed with a `trap ... EXIT` rather than a trailing `rm`,
  which `bash -e` skips when sign-release.sh exits non-zero - leaving the PEM on
  the runner for the rest of the job.
- The AUR host-key cross-check reported "does not match the pinned one" for any
  network failure, because ssh-keyscan's empty output was piped straight into
  grep and its exit code discarded. An alarming message for the wrong reason is
  worse than none; an unreachable host now warns, a DIFFERENT key still stops
  the release.
- docs/release-signing.md said to run sign-release.sh on local files. Rust
  release builds are not bit-reproducible, and the script hashes whatever it is
  given into the .meta sidecar, which the client then enforces against the bytes
  it downloaded - so signing a fresh build produces a release that fails
  verification for everyone. It now downloads the published assets first, and
  carries a runbook for the two failure states, including that "Re-run all jobs"
  reports green while skipping every downstream job.
@MotherSphere
MotherSphere merged commit 54305c4 into main Aug 27, 2026
5 checks passed
@MotherSphere
MotherSphere deleted the fix/release-draft-hold branch August 27, 2026 17:15
MotherSphere added a commit that referenced this pull request Aug 27, 2026
v0.10.0 shipped as a tagged, published, EMPTY release. This is why, and what
stops it recurring.

The draft-hold step added in #70 runs in the `release-please` job, which has no
`actions/checkout`. `gh` infers the repository from the git remote, found none,
and died with "fatal: not a git repository". The job failed, so build, sign and
aur all skipped - and because release-please had already published the release,
/releases/latest moved to a v0.10.0 with zero assets. Every running Colony was
offered an update that does not exist, until the release was re-drafted by hand
and latest fell back to v0.9.2.

aur-publish.yml already carries the warning for exactly this ("`-R` is not
optional here. This job has no `actions/checkout`"), and the new call was added
without heeding it. Every `gh` call in the workflow now passes `-R` explicitly,
so where a job sits can no longer decide whether it works.

The second half is that the pipeline had no way to finish a release once
release-please had emitted its one `release_created`. "Re-run all jobs" makes
release-please report nothing to do and every downstream job skip while the run
reports green, which looks like a successful recovery and is the opposite of
one. So a failure anywhere left a tagged release permanently unfinishable except
by signing four binaries by hand.

`workflow_dispatch` with a tag input fixes that. A new `target` job resolves
which tag the run is for - from release-please on a merge, from the input on a
dispatch - and build, sign and aur read it instead of reaching into
release-please's outputs. Both checkouts pin `ref:` to that tag, because the
default ref on a dispatch is the branch, and rebuilding from a different commit
than the tag names would produce binaries the .meta sidecar then binds to a
version they were not built from.

Three things an adversarial review of this change caught before it merged:

- The `target` job read `needs.target.outputs.tag` - itself. The mechanical
  rewrite that repointed every consumer at the new job also rewrote the one line
  that was supposed to be the SOURCE. On a dispatch it is harmless because the
  input fills in, so the planned v0.10.0 recovery run would have gone green
  while every future merge failed to resolve a tag and left exactly the empty
  tagged draft this commit exists to prevent.
- `if: always()` on `target` would also have run it when release-please FAILED,
  removing the brake that stopped the v0.10.0 incident at an empty draft rather
  than a broken build. `!cancelled() && !failure()` still runs when
  release-please is SKIPPED, which is what a dispatch does.
- The uploader has overwrite_files on by default, so dispatching against an
  already-published tag would delete and replace its live binaries and only then
  hit the draft assertion - leaving published assets whose .sig and .meta
  describe bytes that no longer exist. Since verification is fail-closed, that
  breaks every install of that version. `target` now refuses any tag whose
  release is not a draft, before anything is uploaded.
MotherSphere added a commit that referenced this pull request Aug 27, 2026
…#71)

v0.10.0 shipped as a tagged, published, EMPTY release. This is why, and what
stops it recurring.

The draft-hold step added in #70 runs in the `release-please` job, which has no
`actions/checkout`. `gh` infers the repository from the git remote, found none,
and died with "fatal: not a git repository". The job failed, so build, sign and
aur all skipped - and because release-please had already published the release,
/releases/latest moved to a v0.10.0 with zero assets. Every running Colony was
offered an update that does not exist, until the release was re-drafted by hand
and latest fell back to v0.9.2.

aur-publish.yml already carries the warning for exactly this ("`-R` is not
optional here. This job has no `actions/checkout`"), and the new call was added
without heeding it. Every `gh` call in the workflow now passes `-R` explicitly,
so where a job sits can no longer decide whether it works.

The second half is that the pipeline had no way to finish a release once
release-please had emitted its one `release_created`. "Re-run all jobs" makes
release-please report nothing to do and every downstream job skip while the run
reports green, which looks like a successful recovery and is the opposite of
one. So a failure anywhere left a tagged release permanently unfinishable except
by signing four binaries by hand.

`workflow_dispatch` with a tag input fixes that. A new `target` job resolves
which tag the run is for - from release-please on a merge, from the input on a
dispatch - and build, sign and aur read it instead of reaching into
release-please's outputs. Both checkouts pin `ref:` to that tag, because the
default ref on a dispatch is the branch, and rebuilding from a different commit
than the tag names would produce binaries the .meta sidecar then binds to a
version they were not built from.

Three things an adversarial review of this change caught before it merged:

- The `target` job read `needs.target.outputs.tag` - itself. The mechanical
  rewrite that repointed every consumer at the new job also rewrote the one line
  that was supposed to be the SOURCE. On a dispatch it is harmless because the
  input fills in, so the planned v0.10.0 recovery run would have gone green
  while every future merge failed to resolve a tag and left exactly the empty
  tagged draft this commit exists to prevent.
- `if: always()` on `target` would also have run it when release-please FAILED,
  removing the brake that stopped the v0.10.0 incident at an empty draft rather
  than a broken build. `!cancelled() && !failure()` still runs when
  release-please is SKIPPED, which is what a dispatch does.
- The uploader has overwrite_files on by default, so dispatching against an
  already-published tag would delete and replace its live binaries and only then
  hit the draft assertion - leaving published assets whose .sig and .meta
  describe bytes that no longer exist. Since verification is fail-closed, that
  breaks every install of that version. `target` now refuses any tag whose
  release is not a draft, before anything is uploaded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant